An Audit with the Actual Data
Longitudinal Data From the KETO-CTA Study: Plaque Predicts Plaque, ApoB Does Not
Released by organization funding the study:
https://citizensciencefoundation.org/keto-cta/
All code, figures, data, studies available at:
Figure 1B from Study
Figure 1B from Data
Individual Change in Plaque Volume
(B) The red line represents the median change (0.8%), and the shaded area represents the IQR (0.3%-1.7%).
Figure 1A from Study
Figure 1A from Data
Individual Change in Plaque Volume
(A). The red line represents the median change (18.9 mm3), and the shaded area represents the IQR (9.3-47.0 mm3).
Figure 2F from Study
Figure 2F from Data
Changes in Total Plaque Score vs Coronary Artery Calcium
(C, F) Only CAC is associated with changes in NCPV and TPS. The regression line was fitted with the function “lm,” which regresses y~x, and the shaded area represents the standard error.
4 Simple Linear Regression Assumptions
3 are tested with data
Linearity: between the predictor and the outcome
Constant variance (homoscedasticity) of residuals
Normally distributed residuals
These linear assumptions are quantifiable and objectively testable.
“all linear model assumptions were corroborated with the R function
performance::check_model.”- Direct quote from the study
Actual Linear Model Assumption Tests
| Model | β | Linearity | Constant Variance | Residual Normality |
|---|---|---|---|---|
| ΔNCPV ~ CACbl | β = 0.18 p = <0.001 |
Violation p = 0.031 |
Violation p = 0.001 |
Violation p = <0.001 |
| ΔNCPV ~ NCPVbl | β = 0.25 p = <0.001 |
OK p = 0.198 |
Violation p = <0.001 |
Violation p = <0.001 |
| ΔNCPV ~ PAVbl | β = 5.48 p = <0.001 |
Borderline p = 0.050 |
Violation p = <0.001 |
Violation p = <0.001 |
| ΔNCPV ~ TPSbl | β = 7.37 p = <0.001 |
OK p = 0.132 |
Violation p = <0.001 |
Violation p = 0.001 |
Objective tests show all 4 models failed at least 2 tests.
- quote from letter to the editor
- quote from the reply to a letter to the editor
“residual plot evaluation can also be subjective”
- quote from the reply to a letter to the editor
- quote from the reply to a letter to the editor
Visual checks are interpretive, but these assumptions objectively testable with quantifiable methods.
This is Statistics 101.
MASS::rlm”
check_model output:
“CONCLUSIONS In lean metabolically healthy people on KD, neither total exposure nor changes in baseline levels of ApoB and LDL-C were associated with changes in plaque.”
“there is no association between NCPV vs LDL-C or ApoB and TPS vs LDL-C or ApoB.”
| Abstract claim component | Model | Model reported |
|---|---|---|
| Δ-plaque vs ΔLDL-C | Δ-NCPV ~ ΔLDL-C | Not reported |
| Δ-plaque vs LDL-C exposure | Δ-NCPV ~ LDL-C exposure | Not reported |
| Δ-plaque vs LDL-C baseline | Δ-NCPV ~ LDL-C baseline | Not reported |
| Δ-plaque vs ΔApoB | Δ-NCPV ~ ΔApoB | Reported |
| Δ-plaque vs ApoB exposure | Δ-NCPV ~ ApoB exposure | Not reported |
| Δ-plaque vs ApoB baseline | Δ-NCPV ~ ΔApoB | Reported |
| N/A | NCPV_final ~ LDL-C exposure | Reported (NCPV_final, PAV_final) |
“Results Neither change in ApoB …, baseline ApoB, nor total LDL-C exposure … were associated with the change in noncalcified plaque volume (NCPV) or TPS.”
“Neither … change in ApoB nor the ApoB level … were associated … with TPS (Figures 2D and 2E, Table 3).” - “changes in and baseline levels of ApoB were not associated with changes in NCPV or TPS”
Figures 2D–2F are Δ-TPS (outcome) panels (vs ΔApoB, ApoB, CAC_bl)
Table 3 has no Δ-TPS models
No Δ-TPS ~ LDL-C, LDL-C exposure, or ApoB model results anywhere.
rscale is not Moderate“Bayes factors were calculated…with default settings and an ~
rscale value of 0.8to contrast a moderately informative prior with a conservative distribution width (to allow for potential large effect sizes) due to the well-documented association between ApoB changes and coronary plaque changes.”
Calling rscale = 0.8 “moderately informative” is inaccurate.
R package docs: “medium”, “wide”, “ultrawide” = 0.354, 0.5, 0.707
rscaleCont = 0.8 is wider than “ultrawide” → a very diffuse prior that places substantial mass on very large effectsFixed description:
“We used a very wide prior on coefficients (rscale = 0.8, wider than the package’s”ultrawide”), which places substantial prior mass on very large effects. This diffuse prior penalizes small-to-moderate effects, requiring substantially stronger evidence than under the ‘wide’ or ‘medium’ defaults to support them.”
“LDL-C exposure on a KD was calculated by summing the products of the reported days on a KD prior to study commencement and baseline LDL-C on a KD plus the study follow-up days by their final LDL-C.”
\[ \text{LDL-C}_{\text{exp}} = Days_{\text{KD}}\cdot LDL_{\text{baseline}} \;+\; Days_{\text{follow-up}}\cdot LDL_{\text{final}} \]
“LDL-C exposure on a KD was calculated by summing the products of the reported days on a KD prior to study commencement and baseline LDL-C on a KD plus the study follow-up days by their final LDL-C.”
\[ \text{LDL-C}_{\text{exp}} = Days_{\text{KD}}\cdot LDL_{\text{baseline}} \;+\; Days_{\text{follow-up}}\cdot LDL_{\text{final}} \]
“Estimated lifelong LDL-C additionally included the product of age upon commencing a KD and pre-KD LDL-C.”
\[ \text{Life-LDL-C}_{\text{exp}} = Days_{\text{KD}}\cdot LDL_{\text{baseline}} \;+\; Days_{\text{follow-up}}\cdot LDL_{\text{final}} \;+\; \boldsymbol{\big( Age_{\text{at-KD-start}}\cdot LDL_{\text{pre-KD}} \big)} \]
“Estimated lifetime LDL-C exposure was only a significant predictor of final NCPV in the univariable analysis but lost significance when age was included as a covariate (Table 3). Both age and lifetime LDL-C exposure lost significance when baseline CAC was included in the model (Table 3).”
A mediation analysis:
This is not a mediation analysis.
“Estimated lifetime LDL-C exposure was only a significant predictor of final NCPV in the univariable analysis but lost significance when age was included as a covariate (Table 3). Both age and lifetime LDL-C exposure lost significance when baseline CAC was included in the model (Table 3).”
They ran (reported) three regressions in sequence.
Conclusion: after adjusting for baseline CAC, neither age nor lifetime LDL-C predicts NCPV_final.
- CAC explains the association; age / lifetime LDL-exposure don’t matter.
Changes in p-values come from collinearity due to Age being embedded in lifetime LDL-C exposure.
Age is a confounder/proxy for exposure duration, not a mediator.
CAC association just shows baseline disease predicts future disease.
They did NOT report NCPV_final ~ lifetime LDL-C exposure in the analysis.
“Linear models on the primary (NCPV) and secondary outcomes were univariable”
All conclusions based on independent univariable change score models
Δ-NCPV ~ ApoB; Δ-NCPV ~ NCPV_baseline
Consensus in biostatistics: use ANCOVA style model, follow-up ~ baseline + covariates for pre/post analyses.
NCPV_follow-up ~ NCPV_baseline + ApoB_baseline + age + sex
ΔNCPV ~ ApoB_baseline (unadjusted change score)
Question: Among all participants, is baseline ApoB associated with the raw change in plaque over 1 year, without accounting for where people started or other factors?
Slope interpretation: Average difference in change (mm³) per 1 mg/dL higher ApoB, unadjusted.
Clinically limited: Doesn’t condition on starting plaque, age, sex, BP; easily distorted by baseline differences and extra noise in change scores.
It doesn’t tell an individual with given baseline/risk profile what their 1-year plaque will be.
NCPV_follow-up ~ NCPV_baseline + ApoB_baseline + age + sex (ANCOVA)
Question: Among people with the same starting plaque, age, sex, and BP, do those with higher ApoB tend to have more plaque at 1 year?
Slope interpretation: Expected difference in 1-year NCPV (mm³) per 1 mg/dL higher ApoB, holding baseline NCPV and covariates fixed.
Clinically relevant: What a patient wants to know; “Given where I start and my risk factors, does higher ApoB mean more plaque next year, and by how much?”
Optionally test ApoB × NCPV_baseline to allow baseline-dependent progression.
Pre-print title: Plaque Begets Plaque, ApoB Does Not; Longitudinal Data From the KETO-CTA Trial
Published title: Longitudinal Data From the KETO-CTA Study; Plaque Predicts Plaque, ApoB Does Not
In text: find and replace ‘begets’ with ‘predicts’.
Description of main figure, and primary outcome:
“Most participants presented with stable NCPV (Figures 1A and 1B), with 1 participant exhibiting a decrease in NCPV (Figures 2A to 2C) and 6 participants showing decreases in TPS scores over 1 year”
Removed and replaced with:
“The median change in NCPV was 18.9 mm3 (IQR: 9.3-47.0 mm3) and the median change in PAV was 0.8% (IQR: 0.3%-1.7%).”
In CONCLUSIONS section: “In an exploratory analysis, changes in and baseline levels of ApoB were not associated with changes in NCPV or TPS…”
“exploratory analysis” added in published version. Only mention in entire paper.
Table 1 median (Q1–Q3) PAV at baseline changed from 1.25% (0.5–3.6) to 1.6% (0.5–4.9).
Record of pre-print is unavailable. All links go to new version. It is available here.
| Where | Paper text (claim) | Issue |
|---|---|---|
| Figure 1 caption | “…the shaded area represents the IQR …” | Imprecise. IQR is a single number; shaded band is 25th–75th percentile band at baseline and 1-year. |
| Figure 1 caption | “(9.3–47.0 mm³)” | Wrong context/unstated. Those numbers are IQR of ΔNCPV |
| Figures 2D–F | “Change in ApoB, baseline ApoB, and baseline CAC vs NCPV.” | Axes/legend appear to be TPS, not NCPV → label/data mismatch |
| Table 3 caption | “R² = squared correlation coefficient (explained variability).” | Reports negative values, must be adjusted R² |
| KD Duration Table 1 vs Table 3 | KD duration mean 1,642.7 days (≈4.5 y) vs caption “LDL-C exposure … mean 5.7 y.” | Inconsistent durations. |
| KD Duration Abstract vs Table 1 | LDL-C exposure median 1,302 d (984–1,754) vs KD duration median 1,427 d (1,002–1,938). | Metrics conflated. exposure should be in units like mg/dL·days, not days. |
| Table 1 (Total cholesterol) | Median 338, IQR (301–337). | Impossible IQR (Q3 < median) |
| Baseline characteristics only | Baseline table provided. | No follow-up characteristics table |
| Terminology (multiple) | “changes in baseline levels” | Baseline doesn’t change. |
| Software statement | “using R 4.0.3 (2020)… with the last available version for Sept 2024.” | Unclear what version was used. |
| Central illustration (quote) | “Neither total exposure nor changes in baseline levels of ApoB and mg/dL were associated… Baseline plaque but ApoB was not associated…” | Incoherent/grammatical errors |
| Main text (Results/Discussion) | “…baseline values magnify their percentual changes…” | percent change or percentage-point change |
“The median change in NCPV was 18.9 mm3 (IQR: 9.3-47.0 mm³) and the median change in PAV was 0.8% (IQR: 0.3%-1.7%). Compared to baseline, these represent a 43% and 50% change, respectively.”
This isn’t the standard subject-level percent change.
They computed a ratio of median change to baseline median: \(\frac{\operatorname{median}(NCPV_{1y}-NCPV_{bl})}{\operatorname{median}(NCPV_{bl})}\times 100\%\)
Standard percent change: \(\operatorname{median}\!\left(\frac{NCPV_{1y}-NCPV_{bl}}{NCPV_{bl}}\right)\times 100\%\)
“What was the typical relative change per subject?”
Standard percent change vs. reported ratio-of-medians:
| Outcome | “ratio-of-medians” | Median % change | Mean % change |
|---|---|---|---|
| NCPV | 43% | 49.2% | 81.4% |
| PAV | 50% | 47.3% | 80.7% |
Their “percent change” metric is mislabeled, nonstandard and not well explained.
Plaque Metrics Summary Table
| Baseline, follow-up, and paired change in plaque metrics at 1 year | |||||||
|---|---|---|---|---|---|---|---|
| Metric | N1 |
Median (Q1–Q3)
|
Mean ± SD
|
||||
| Baseline | Follow-up | Change2 | Baseline | Follow-up | Change2 | ||
| Noncalcified plaque volume (mm³) | 100 | 44.0 (15.5–102.2) | 66.0 (25.2–163.2) | 18.8 (9.3–46.6) | 75.9 ± 88.3 | 107.4 ± 112.7 | 31.5 ± 31.5 |
| Coronary artery calcium score | 100 | 0.0 (0.0–54.2) | 0.0 (0.0–56.0) | 0.0 (0.0–5.0) | 50.3 ± 100.9 | 58.8 ± 120.4 | 8.5 ± 25.5 |
| Percent atheroma volume (%) | 100 | 1.6 (0.5–5.0) | 2.6 (1.0–6.8) | 0.8 (0.3–1.7) | 3.2 ± 3.8 | 4.4 ± 4.6 | 1.2 ± 1.2 |
| Total plaque score | 100 | 0.0 (0.0–2.2) | 1.0 (0.0–3.0) | 0.0 (0.0–1.0) | 1.7 ± 2.6 | 2.2 ± 3.0 | 0.5 ± 1.1 |
| Calcified plaque volume (mm³) | 100 | 0.3 (0.0–19.8) | 0.9 (0.0–24.2) | 0.5 (0.0–4.2) | 18.7 ± 37.4 | 22.7 ± 42.5 | 4.0 ± 7.4 |
| 1 N = number of paired scans per metric. | |||||||
| 2 Change = follow-up − baseline, summarized across participants. | |||||||